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ABSTRA.CT 

Two major topics are covered in this paper — the 
evaluation of instructional programs in general and the evaluation of 
computer- assisted instructional (CAI) course material during initial 
preparation. The first half of the paper reviews significant 
literature relating to instructional program evaluation and 
distinguishes between formative and summative evaluation. It then 
describes five models of evaluation, including three which specify 
procedures for the initial preparation of instructional programs, a 
foinrth which represents a model for summative evaluation by an 
outsider, and a fifth which depicts several classes and data to be 
gathered during a comprehensive evaluation. The other major section 
of the paper deals with matters relating to CAI materials. Literature 
on the formative and summative evaluation of CAI programs is 
reviewed, followed by a discu^iaSion of consequential evaluation and an 
overview of some evaluative criteria. A model for CAI course 
preparation and evaluation is presented and recommendations for the 
inclusion of formative evaluation as an integral part of curriculum 
development are made. (Author/PB) 
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1* Formative and Summative Evaluation 
Authorities have referred to Cronbach's paper entitled "Course Improvement" 
LiJ Through Evaluation^' presented in 1963 as a "classic" (Glass, ^9C3). In this 
paper Cronbach defined evaluation as the "...collection and use of information 
to make decisions about an educational program" (Cronbach, 1963, p, 672). He 
indicated that such information could be used for course improvement, for deci- 
sions about individual stadents, or for administrative regulations. Cronbach 
emphasized the importance of evaluation for the purpose of course improvement: 
The greatest service evaluation can perform is to identify 
aspects of the course where revision is desirable... To be influ- 
ential in course improvement, evidence must become available midway 
in curriculum development, not in the home stretch, when the 
developer is naturally reluctant to tear open a supposedly finished 
body of materials and techniques. Evaluation, used to improve the 
course while it is still fluid, contributes more to improvement of 
education than evaluation used to appraise a product already placed 
on the market (Cronbach, 1963, p. 675). 

Cronbach stated that the analysis of performance on single test items or 
the record of responses to different types of problems could be more informative 
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than an analysis of total scores. He viewed evaluation as; 
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...a fundamental part of curriculum development, not an 
appendage. Its job is to collect facts the course developer 
can and will use to do a better job, and facts from which a 
deeper understanding of the educational process will emerge 
(Cronbach, 1963, p. 683). 

Scrlven (1967) proposed using the terms "formative" and "summative" to 
distinguish betv/een evaluation to improve an instructional program or curriculum 
during Us development and evaluation to determine the worth or effectiveness of 
an instructional program once it had been completed! He suggested that, in 
order to avoid potential clashes between curriculum v^riters and professional 
evaluators, 

...formative evaluators should,. if at all possible be sharply 
distinguished from the summative evaluators, with whom they may 
certainly work in developing an acceptable s'ummative evaluation 
schema, but the format1\/e evaluators should ideally exclude them- 
selves from the role or judge in the summative evaluation (Scriven, 
1967, p. 45). 

Linavall and Cox (1970) point out that, "Once a program has been developed . 
and is fully functioning, the task of a summative evaluator is to describe jus-t 
what that program does or what it is worth" (Lindvall and Cox, 1970, p. 56). 

Scriven maintained that in the early stages of any kind of curriculum project 
general objectives or goals are formulated. These goals, which should not be 
considered absolute commitments, but rather reminders subject to alteration, 
might range from motivational and cognitive goals to the goal of producing a 
marketable program. Scriven declared that these goals were to be themselves 
items for evaluation; performance measured against goals was not to be the only 
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concern of the evaluator. To* him it was "...obvious that if the goals aren*t 
worth achieving then it is uninteresting hov/ well they are achieved" (Scrive.n, 
1967, p. 52). 

Scriven outlined three types of activities which could facilitate both the 
evaluation of the goals and the evaluation of performance measured against those 
goals. These activities are: 

1 Regular reexamination and modification of proposed 
general objectives or goals of the pro-^'ect. 

2. Construction of a test-question pool, which thus becomes 
an "operational version of the goa'^s" (Scriven, 1967, p. 
56) and, as such, also requires regular reexamination and 
modification in light of any changes in the project goals. 

3. External judgments about the consistency of the project 
goals, content, and test-question pool. 

Scriven saw several refinements of the above activities as crucial to formative 

evaluation studies since they could uncover the causes of poor results: 
Lssentidlly, we need to know about the success of three 
connected matching* problems: first, the match between goals 
and course content; second, the match between goals and examina- 
tion content; third, between course content and examination content, 
...Only in this way are we likely to be able to track down the 
real source of disappointing results (Scriven, 1967, p. 59). 
Stolurow in a paper presented at a Council for Exceptional Children 

Special Conference on Instructional Technology commented on the function of 

formative evaluation: 
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It IS the formative evaluation process that results in 
specific revisions of a program to improve its rhetoric, instruc- 
tional effectiveness, and acceptability (Stolurow, 1970, p, 75) • 
In Handbook on For mative and Summati ve Evaluatio n of Student Learning , 
Bloom, Hastings, and Madaus defined evaluation as: 

• •.the systematic collection of •evidence to determine 
whether in fact certain changes are taking place In the learners 
as well as to dete^^mine the amount or degree of change In indi- 
vidual students (Bloom, Hastings, an^ Madaus, 1971, P-3). 
They distinguished between .u ^ive and iummative evaluation on the basis of 
purpose, time at which evaluation occurs, and .level of generalization sought 
by the items in the examination usr>d to collect data for the evaluation'* (Bloom, 
Hastings, and Madaus, 1971, 611. 

We have chossr ' ^v-^ t^m surumative evaluation'* to indi- 
cate the type of e^^,. uation used at the end of a term, course, 
or program for purposes of grading, certification, evaluation 
0^ nrogress, or research on the effectiveness of a curriculum.-. 

. Formative evalijation Is for us the use of systematic eval- 
uation in the process of curriculum construction, teaching and 
learning for the purpose of improving any of these three processes 
(Bloom, Hastings, and Madaus, 1971, p. 117). 
In the Preface to their book the authors explained that their interest is the 
improvement of student learning, as the title of their book would indicate. 

Airasian also focused on formative evaluation for the improvement of 
student learning. He stated that fc^^ative evaluation "...seeks to identify 
learning weaknesses prior to the completion of instruction on a course segment" 



(Airasian, 1971, p» 79)» .He sunmiarized differences between formative and 
summative evaluation by indlcatinq the verb tense used with each term: 
*\ . .Formative evaluation provides data about how students are changing. 
Summative evaluation is concerned with how students ha^e changed. (Airasian, 
197U 0. 78). 

2. Models tor Evaluation of instructional Programs 
In the following discussion five rodelr, for the evaluation of instructional 
orograms will be described. The first three specify procedures for the initial 
oreparation of instructional oroorams Tlie fourth reoresents a model for 
summative evaluation conducted by an outside evaluator. The fifth model depicts 
several classes of data to be gathered when conducting a comprehensive evaluation* 
Mode 1 I > Stake '( 1967) indicated tiat two main types of information are necessary 
for the evaluation of e^ii ationul Dror<:in:v. The ti i^ t^r^f, the intents and 
outcomes, and the secono is oersunal judgy^nts db lo the quality and appropriate- 
ness of the intents and outcomes* 

In another article Stake (1967) explained what his proposed evaluation program 
would involve* Descriptions of what intended antecedents or entry behaviors 
were exoected, what intended transactions or instructional processes were 
planned, and what outcomes were anticipated would be evaluated for their logical 
relationship to edch other* Then the descriotions of what actually happened would 
be examined to determine if what was intended actually occurred (see Figure 1)* 
Finally, judgments of the value of the instructional orogram would be based on 
absolute standards reflected by the evaluator's personal judgment and on relative 
standards reflected by comparison of the oarticular program to alternative programs 
(see Figure 2)* Program designers would prepare a rationale stating the basic 
ourdose and phi losoohical background of their program which would assist the 
evaluators. 



stake posed five questions which he felt should be answered prior to the 
initiation of evaluation procedures: 

"1. Is this evaluation primarily descriptive, primarily judgmental, 
or both descriptive and judgmental? 

2. Is this evaluation to emphasize the antecedent conditions, the 
transactions or the outcomes alone, or a combination of these,, 
or their functional contingencies. 

3. Is this evaluation to indicate the congruence between what is 
\ intended and what occurs? 

4. Is this evaluat':on 'u i;.. undertaken within a single program or 
as a comparison between two or more curriculum programs? 

5. ^, Is this evaluation intended more to further the development of 

curricula or to \^e'>o .h.^r^e diiono available curricula?" (Stake, 
1967, D, 539)- 

Stake here did not report the sequence in which the steps of -his process 
\70uld be followed, nor did he illustrate his process. No reports' of projects 
in which his evaluation procedures had been used were located in the literature. 
lYoch has Suggested an expansion of the Stake model to increase the 

information yield about new educational products. 

Model II . Briggs 1,1970) in his monograph entitled Handbook of Procedures for 
the Development of Instructional Systems presented a model for the preparation 
of new instri'Ctional course material. His model, which encompasses course 
design, development, and evaluation, provides for the deliberate selection 
or creation of instructional ^materials on the basis of both learner character- 
istics and the nature of the competencies which the course is supposed to 
develop, as well as on the basis of the characteristics of the material 
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alternatives (see Figure 3). The monograph is devotea to the design pha^-a 
of Briggs' model. Briggs stated that formative evaluation procedures would 
s,tart during the development and evaluation phases which he discussed briefly, 
fie listed suggestions for formative evaluation which could be followed sub- 
sequent to what he called "formative design" steps taken during the develop- 
ment of first-draft materials in Steps 1-6. Briggs defined formative design 
as "...the use of nerformance tests (empirical data) for making the necessary 
decisions long before first-draft materials are ready for try-out" (Briggs, 
1970, p. .173). In his critique of his model written after he and his graduate 
students had examined twenty other models for instruction drawn from military, 
industrial, educational, and governmental settings, Briggs observed several 
limitations in his model: 

The model/is somewhat limited from the point of view of 
planning the/integration of materials, snace, teachers, and 
learners into an administrative and management system for the 
operation of the learning envi ronment* . J'lhereas the model may 
be inadequate for skills of inquiry needed for advanced types 
of problem solving, it is clearly, useful as a guide for planning 
instruction at many of the less advanced levels (Briggs, 1970, p. 
185). 

Model III . Baker and Schutz declared, "Most instruction is dispensed, not 
developed" (Baker and Schutz, 1971, p. xv)» They characterized instructional 
development as "...essentially a cyclical process, ...a team effort, and ... 
user-oriented" (Raker and Schutz, 1971, pp. xv-xvi). They viev/ed an adequate 
instructional development program as one giving consideration to five program 
systems: Instructional, Training, Installation, Accountability, and Modification. 
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The Instructional system in the opinion of Baker and Schutz is the key system 
from which specifications for the other four systems are derived. All systems 
share common characteristics and are closely interdependent, although each system 
has a distinct function within the total development program. 

Baker and Schutz listed seven components of their instructional development 
cycle which cuts across all five program systems and system characteristics. 
These components are: 

1 . Formation 

2. Instructional Specification 

3. Item Tryout 

4. Product Development 

5. Product Tryout 

6. Product Revision 

7. Operations Analysis (Baker and Schutz, 1971, p. 131). 

For each of the seven components in the development cycle Popham and Baker 
(1971) specified general rules (see Appendix A). In addition, Popham described 
principles demonstrated to be effective in following the rules for activities 
within each component of the development cycle. These principles are: 
"1. Provide relevant practice for the learner. 

2. Provide knowledge of results. 

3. Avoid the inclusion of i rrelevancies . 

4. Make the material interesting" (Popham, 1971, p. 171). 

To produce the Interest required in the last principle listed above, Popham 
urged the deliberate use of variety, humor, game-type situations, suspense, 
and format variations. 

Model IV . Glass (1972) applied a prototype evaluation format to the appraisal 
of an educational product already on the market, an instructional 100-foot 
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cassette tape recording of a presentation entitled "Evaluation Skills" given by 
Dr. Michael Scriven. The model covered the following items: 

1. Product description 

2. Goals evaluation 

- 3. Clarification of point of entry of the evaluator 

a. Irreversible decisions 

b. Reversible decisions (Enter the evaluator.) 

4. Trade-Offs 

5. Comparative cost analysis 

6. Intrinsic (secondary) evaluation 

a. Technical quality 

b. Content evaluation 

c. Utilization of uniqueness of medium 

d. Survey of availability 
7* Outcome (primary) evaluation 

^ ,8. Summative judgments and recommendations 

9. Circumstances modifying the summative judgments (scope and value claims) 
Glass's prototype model was prepared for the outside evaluator to follow 
in appraising a finished instructional product. 

Model V . The CIPP model was developed by Stuff lebeam and his associates at the 
Ohio State University. It might be regarded as a heuristic model for generating 
data classes. The CIPP Model has four components: Context, Inputs, Process, 
and Product (Stuf flebeam, 1970). 

Those four components should help provide relevant data to decision-makers. 
^ First: Context - How-will CAI fit into the overall plan of operation and 
the goals of the organization? How will present personnel react? What will 
their roles be? Will there be union problems? What about scheduling? Space? 
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Secorid: Input - What do the students bring to the learning situation? What 
are the desired entry behaviors? What prerequisite knowledge or skills are required? 

Third: Process - What is the quality of interaction between student and 
system? How are individual differences accounted for? How well does the system 
match individual students with different instructional strategies? What are the , 
testing procedures? 

Finally: Product - Does the system work? Do all students nrieet all objectives? 
Are all relevant objectives covered? Are irrelevant objectives included? How 
much time is required? Are the students well prepared for whatever follows the 
program? How is student behavior changed? Is belYavior changed in this context 
only? How long is the behavior maintained? Do students like the program? How 
about other personnel? 

There is an extensive literature related to this model. Readers are 
referred to Dr. Don Stufflebeam, at the Evaluation Center, Ohio State University, 
for more information » 

3. Summary 

Authorities have distinguished between formative and summative evaluation 
dfid have developed models for authors of instructional programs to follow. Stake's 
plan for evaluation provides a general outline for instructional development 
projects which make evaluation an integral part of the project* Briggs' model 
places emphasis on the selection of available or the design of new materials in 
order for students to reach instructional 'objectives. Glass reports the results 
of his having used a model appropriate for the evaluation of finished products. 
Baker and Schutz outline a practical program for the newcomer in program develop- 
ment to follow. 

Factors influencing the choice of a particular model would seem to include 
the purpose for and the scope of the evaluation, the point at which the evaluation 
is to be initiated, and the person to whom the task of evaluation is assigned. 



B. Evaluation of Computer-Assisted Instruction 
Course Material During Initial Pre,paration 

Rogers (1968) several years ago reviewed problems in CAI and observed that 
lack of quality CAI course material constituted a major, problem. He called 
attention to the need for evaluated course materials. 

Cartwright has identified recent trends in curriculum evaluation: evalua- 
tion is becoming acceptable and broader in base; as the contribution that 
formative evaluation can make to curriculum development receives greater 
recognition, there is a corresponding decrease in emphasis on summative evalua- 
tion; and in spite of th^s recognition, "...the large majority of CAI publications 
and papers that have become available in the last two years still are reporting 
summative evaluation activities..." (Cartwright , 1 971,5 p. 2). 

1. Formative and Summative Evaluation 
Cartwright and Mitzel (1971) described both the formative and summative 
procedures they follov/ed during the preparation of a three-credit CAI course 
designed for regular classrocm teachers primarily in rural. areas entitled 
"Earl;/ Identification of Handicapped Children." During the formative evalua- 
tion procedures, which covered approximately six months, fifteen students took 
the course while a proctor observed and recorded any student comments and 
program bugs. Technical problems went to the programer and content problems 
were given to the author who made necessary changes. Once all revisions had 
been organized, the course was revised and a second pilot group of fifteen 
students took the course unattended by a proctor. In addition, two graduate 
students in special education completed the course and submitted their evalua- 
tion reports. Then, 115 inservice teachers completed the course for college 
credit as a part of Penn Staters Continuing Education program. Extensive 
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revisions were made as a result of the analyses of the responses, requests 
for assistance, and response latencies collected from these students. Finally, 
300 more students completed the course and additional revisions were made as a 
result of data collected on those students. All told, over 333,000 student 
responses were analyzed during*' the formative evaluation. 

To conduct a suimiative evaluation of the course, on-campus students who 
registered for "Introduction to the Education of Exceptional Children" were 
randomly assigned to conventional instruction (CI) and to CAI, Objectives for 
both courses were the same; in fact, tlie teacher of CI class had been one of 
the CAI course authors. Using time to complete the course and score on the 
75-item final exam as variables, the authors reported that analyses of their 
data indicated the CAI students (n=27) scored significantly higher than CI 
students (n=87) on the final exam and completed the course in twelve hours 
less time than the CI students. 

Confer (1970) repo'^ted another summative evaluation of a, CAI course 
designed to teach general math. Students, all repeaters in general math, were 
randomly assigned to regular class instruction and to CAI instruction during 
a Summer school session. Performance at the end of instruction in computation 
and problem-solving was measured with the Stanford Achievement Test (SAT), 
Analysis of covariance indicated no significant differences betu/een the two 
groups on SAT scores. Confer concluded that his results neither confirmed 
nor rejected CAI as a method of instruction. Among his recommendations was 
the need for an analysis of all students' responses to help determine necessary 
changes in the CAI general math course. 

In a speech at the Association for the Development of Instructional Systems 
in 1971 5 Cartwright stated: 

It is unlikely that summative evaluation per se will improve 
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the quality of instruction. Formative evaluation, however, is 
a model that can be used to i>orove the quality of instruction. 
...It seems to me that criterion-referenced instruction as a goal 
and formative evaluation as a method is the way to go at this 
point in time in the devel .t of CAI (Cartwright, 1971, p. 9). 

2. Consequential Evaluation 
Glass (1969) has described consequential evaluation as follows: "Con- 
sequential evaluation is evaluation of the affects (sic) of the program.*, 
(the consequences of the program)" (Glass, 1969, p. 5). Formative evaluation 
deals with data collection for the purposes of improving the course and 
summative evaluation is concerned with the "comparative worth or effectiveness 
of competing programs" (Glass, 1969, p. 5). Consequential evaluation is 
concerned with a more slio.^ery criterion long range performance or behavior 
of the learners who have taken the CAI program. In the case of CAI programs in 
teacher education, we ure concerned with the performance of teachers in the 
classroom and, ultimately the behavior of the children in those classrooms. 
Good consequential evaluation studies are quite expensive to carry out and 
quite difficult to manage. 

3. Criteria for Evaluation of CAI 
Seltzer has written: 

V/hat the computer can and cannot do is a matter of research 
and fact. What the computer should and not do in instruction is 
based on value judgments.. .(Seltzer, 1971, p. 373). 
Seltzer suggested that, in order to be in a- position to make value judgments, 
criterion statements should be drawn up for use in evaluating the selection 
of the computer to assist in any particular instructional process. The 
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criterion statements Seltzer proposed are: 

"1. If thfe computer poses a unique solution to an important 
problem in the instructional process, then it should be 
used regardless of the cost involved. 

2. If the computer is more efficient or effective and the 
cost of its use to instruct is minimal, then it should 
be used. And conversely, 

3. If the cost of development and use of the computer in 
instruction is relatively high with the relative efficiency 
or effectiveness only marginal, then the computer should 
not be. used in the instructional process" (Seltzer, 1971, 
p. 375). 

These criterion statements look at CAI cost, effectiveness, and efficiency 
in comparison to alternative means of instruction. They could logically be 
considered during the initial design of a, proposed CAI application to instruc- 
tion before much instructional material had been developed. 

4 Model for CAI Course Preparation and Evaluation 
Bunderson constructed a prescriptive model for the design of CAI course 
material. He explained the circumstances which prompted his effort: 
The instructional design model described in this chapter 
was originated to provide management and quality control for 
curriculum development, to provide a bridge between the curriculum 
development and basic research activities of the laboratory, and 
to serve as a focus for teaching students and others how to design 
quality CAI programs. Its development was influenced by the author's 
attempts to adjust to a joint appointment between educational 
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psychology and compi;ter science and to communicate with 
staff members and students from both fields (Bunderson, 
1970, p. 46). 

Bunderson discussed the activities to be performed by the instructional 
designer, their approximate sequence, and the product of each activity. The 
design activities in the sequence Bunderson outlined are: 

1. Intent and justification 

a. Write societal needs. 

b. Write institutional needs; 

c. Write program goals. 

1. Describe job requirements. 

2. Describe student population. 

3. Describe institutional constraints. 

d. Write justification for CAI. 

2. Instructional design: analysis 

a. Derive operational requirements from goals. 

1. Derive terminal objectives. 

2. Set entering performance standards. 

3. Consider effect of constraints on program design. 

b. Behavioral analysis 

1. Obtain intermediate objectives through analysis 
of terminal objectives. 

2. Construct learning hierarchy. 

c. Analysis of learner traits 

3. Instructional design: synthesis 
a. Specify interface. 

ERLC 
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1. Display and response devices. 

2. Representation. 

b. Construct individualizing flow chart. 

1. Hierarchy-based gating mechanisms. 

2. Trai t-by-treatment branches. 

3. Continuously adaptive mechanisms. 

c. Write working draft. 

K Construct curriculum-embedded tests controlling " 
major flow. 

2. Write steps and describe format of steps. 

4. Produce program materials. 

a. Code from author's draft. 

b. Produce media. 

c. Debug code and proof media. 

5. Evaluate and revise. 

a. Editorial evaluation. 

b. Internal empirical evaluation, 
c External empirical evaluation. 

1. Validation testing. 

2. Longitudinal validation. 

6. Use of feedback. 

Return to any previous step as indicated by evaluation, 
revise, and recycle. 

In his discussion of parts of his model, Bunderson observed that the con- 
struction of a learning hierarchy (see above, 2.b.2.) seems ... 
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.•.readily applicable to any cumulative subject matter 
such as mathematics, much of science, and even music. It 
seems less applicable to highly verbal areas (Bunderson, 
1970, p. 56). 

5. Summary 

. A survey of the literature related to both evaluation and CAI reveals that; 
models for formative evaluation are available for use in developing course 
materials, that authorities urge formative evaluation be incorporated into 
initial CAI course development projects, and that to date formative evaluation 
procedures have not been reported in many completed projects ./ In the one 
model designed specifically for CAI course preparation, the author observed that 
some of the activities he outlined were more suitable for. subject matter with 
inherent structure rather than for highly verbal subject matter on which 
several structures might be imposed. 

Authorities continue to stress the need for formative evaluation during 
initial CAI course preparation for purposes of course improvement. They see 
l:ttie information in summative evaluation results that can help authors locate 
course weaknesses or errors. 

The application by a course author of a formative evaluation model to the 
initial preparation of a CAI course would seem strongly indicated. 



APPENDIX A 

^ A REVIEW OF PRODUCT DEVELOPME^JT RULES 
(Baker and Schutz, 1971, pp. 167-68) 

FORMULATION 

F:l. The extensi veness of a proposed product's justification should be 

communsurate with the importance of the product. 
F:2. Excessive time should not be spent in formulation. 
F:3. In justifying the development of the new product, make certain there 

are no competing products of high quality. 
INSTRUCTIONAL SPECIFICATIONS 

IS:1. All instructional objectives should be stated in terms of the 

learner ' s post-i nstructi onal behavior. 
1S:2. En-route and entry behaviors should also be described behaviorally 

in the instructional specifications. 
IS:3, Criteria for judging the adequacy of the learner's response should 

be specified. 

IS:4. A clearly specified method for determining learner affect toward the 

completed instructional product should be specified. 
ITEM TRYOUT 

1T:1. The criterion test must be completely prepared prior to the deve- 

lopment of the instructional product. 
1T:2. Measures of the entry and en-route behaviors should be constructed 

during the item tryout stage. 
1T:3. Prototype, items should not deviate from the behaviors described in 

the instructional specifications. 
IT:4. Prototype items should be tried out with a small number of learners 

first,, later with a larger number of learners. 



PRODUCT DEVELOPMENT 

PD:K Supply the learner with appropriate practice during an instructional 
sequence. 

PD:2. ' The product should provide the learner with the opportunity to obtain 

knowledge of results. 
PD:3. The instructional product should contain provisions for promoting the 

learner's interest in the product. 
PD:4. Avoid the development of an inflexible strategy in approaching 

product development tasks. 
PD:5. If teachers are involved in the instructional process, make their 

participation as replicable as possible. 
PD:6. In general , adopt a "lean" programming strategy. 
PD:7. If the product is to be used in the classr^obm, develop it so that 

teacher attitudes toward the product will be positive. 
PD:8. Selection of the instructional medium should be made in light of the 

desired instructional objectives, intended target population, cost, 

and other relevant considerations. 
PD:9. Tne time devoted to :-he development of the product should be commen- 
surate with the importance of the product, 
PRODUCT TRYOUT 

PT:1. Avoid an extremely small or extremely large number of learners when 

field testing the product. 
PT:2. Verify that the procedures associated with the use of the product 

result in a replicable treatment. 
PT:3. Data from field trials should be efficiently summarized for use by 

those who will revise the product. 
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PT:4. Those involved in field testing the product should collect data; they 

- should not, themselves, engage in drawing inferences from the data. 
PRODUCT REVISION 

PR:1. Base product revisions on legitimate inferences from field test data. 
PR:2. Jhe primary inferences regarding product revisions should be made 

from criterion data. 
PR: 3. Learner response data during the program should be considered a 

valuable source of cues for product improvement. 
PR:4. No loss of face for the .initial developer should be associated with 

revisions of an instructional product. 
OPERATIONS ANALYSIS 

0A:1. Operations analysis should be performed at the conclusion of all 

systematic development of instructional products. 
0A:2- The operations analysis should be written and transmitted to some 

central repository. 
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